System for rendering color images

ABSTRACT

An image is rendered on a display having a limited number of primary colors by (104) combining input data representing the color of a pixel to be rendered with error data to form modified input data, determining in a color space the simplex (208—typically a tetrahedron) enclosing the modified input data and the primary colors associated with the simplex, converting (210) the modified image data to barycentric coordinates based upon the primary colors associated with the simplex and (212) setting output data to the primary having the largest barycentric coordinate. calculating (214) the difference between the modified input data and the output data for the pixel, thus generating error data, applying (106) this error data to at least one later-rendered pixel, and applying the output data to the display and thus rendering the image on the display. Apparatus and computer-storage media for carrying out this process are also provided.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending application Ser. No.16/731,160 filed Dec. 31, 2019, to be issued as U.S. Pat. No.10,771,652, which is a continuation of application Ser. No. 16/280,306,filed Feb. 20, 2019, now U.S. Pat. No. 10,554,854, which is acontinuation of application Ser. No. 15/592,515, filed May 11, 2017, nowU.S. Pat. No. 10,270,939, which claims benefit of provisionalApplication Ser. No. 62/340,803, filed May 24, 2016.

This application is also related to application Ser. No. 14/277,107,filed May 14, 2014 (Publication No. 2014/0340430, now U.S. Pat. No.9,697,778, issued Jul. 4, 2017); application Ser. No. 14/866,322, filedSep. 25, 2015 (Publication No. 2016/0091770); U.S. Pat. No. 9,383,623,issued Jul. 5, 2016 and U.S. Pat. No. 9,170,468, issued Oct. 27, 2015.Other related applications and patent will be discussed below. Theentire contents of these copending applications and patent (which mayhereinafter be referred to the “ECD” patents), and of all other U.S.patents and published and copending applications mentioned below, areherein incorporated by reference.

BACKGROUND OF INVENTION

This invention relates to a method for rendering color images. Morespecifically, this invention relates to a method for half-toning colorimages in situations where a limited set of primary colors areavailable, and this limited set may not be well structured. The methodof the present invention is particularly, although not exclusively,intended for use with color electrophoretic displays

Half-toning has been used for many decades in the printing industry torepresent gray tones by covering a varying proportion of each pixel ofwhite paper with black ink. Similar half-toning schemes can be used withCMY or CMYK color printing systems, with the color channels being variedindependently of each other.

However, there are many color systems in which the color channels cannotbe varied independently of one another, in as much as each pixel candisplay any one of a limited set of primary colors (such systems mayhereinafter be referred to as “limited palette displays” or “LPD's”);the ECD patent color displays are of this type. To create other colors,the primaries must be spatially dithered to produce the correct colorsensation. It is known to effect such spatial dithering by using, forany desired color, only the primary colors at the vertices of atetrahedron which contains the desired color; see, for example:

-   -   Arad, N., Shaked, D., Baharav, Z., & Lin, Q. (1999). Barycentric        Screening and Ostromoukhov, Victor, and Roger D. Hersch.        “Multi-color and artistic dithering.” Proceedings of the 26th        annual conference on Computer graphics and interactive        techniques. ACM Press/Addison-Wesley Publishing Co., 1999.        Both these documents effect dithering by means of a        threshold-array based screening method, which is a simple        dithering method that has been found not to give good results in        ECD patent displays.

Standard dithering algorithms such as error diffusion algorithms (inwhich the “error” introduced by printing one pixel in a particular colorwhich differs from the color theoretically required at that pixel isdistributed among neighboring pixels so that overall the correct colorsensation is produced) can be employed with limited palette displays.However, such standard algorithms are typically intended for use with alimited palette which is “well structured”, in the sense that thedistances in the appropriate color space between the primary colors aresubstantially constant. There is considerable literature on the problemsof designing optimal color palettes that perform well with errordiffusion; see, for example:

-   -   Kolpatzik, Bernd W., and Charles A. Bouman. “Optimized Universal        Color Palette Design for Error Diffusion.” Journal of Electronic        Imaging 4.2 (1995): 131-143.        However, in ECD and similar limited palette displays, in which        the limited palette is defined by the colors capable of being        generated by the system, the limited palette may not be well        structured, i.e., the distances between the various primaries in        the color space may differ greatly from one another.

FIG. 1 of the accompanying drawings is a schematic flow diagram of aprior art palette based error diffusion method, generally designated100. At input 102, color values x_(i,j) are fed to a processor 104,where they are added to the output of an error filter 106 (describedbelow) to produce a modified input u_(i,j). The modified inputs u_(i,j)are fed to a quantizer 108, which also receives details of the palette{P_(k)} of the output device. The quantizer 108 determines theappropriate color for the pixel being considered, given by:

$y_{i,j} = {\arg {\min\limits_{P_{k}}{{u_{i,j} - P_{k}}}}}$

and feeds to appropriate colors to the device controller (or stores thecolor values for later transmission to the device controller). Both themodified inputs u_(i,j) and the outputs y_(i,j) are fed to a processor110, which calculates error values e_(i,j), where:

e _(i,j) =u _(i,j) −y _(i,j)

The error values e_(i,j) are then fed to the error filter 106, whichserves to distribute the error values over one or more selected pixels.For example, if the error diffusion is being carried out on pixels fromleft to right in each row and from top to bottom in the image, the errorfilter 106 might distribute the error over the next pixel in the rowbeing processed, and the three nearest neighbors of the pixel beingprocessed in the next row down. Alternatively, the error filter 106might distribute the error over the next two pixels in the row beingprocessed, and the nearest neighbors of the pixel being processed in thenext two rows down. It will be appreciated that the error filter neednot apply the same proportion of the error to each of the pixels overwhich the error is distributed; for example when the error filter 106distributes the error over the next pixel in the row being processed,and the three nearest neighbors of the pixel being processed in the nextrow down, it may be appropriate to distribute more of the error to thenext pixel in the row being processed and to the pixel immediately belowthe pixel being processed, and less of the error to the two diagonalneighbors of the pixel being processed.

Unfortunately, it has been found that if one attempts to useconventional error diffusion methods such as that shown in FIG. 1 to ECDand similar limited palette displays, severe artifacts are generatedwhich may render the resultant images unusable. For example, in one typeof artifact, (hereinafter called a “transient” artifact) when steppingfrom one input color to a next very different color, the spatialtransient can be so long that the output never settles to the correctaverage even across the size of object being rendered. In another typeof artifact (hereinafter called a “pattern jumping” artifact), for aconstant color input image, the output jumps between two different setsof primaries at a seemingly random position in the image. Although bothsets of primaries should ideally produce output close to the color beingrequested, the resultant output is not robust because small changes inthe system can cause these switching between the two sets and thetexture change at such a jump is also noticeable and unpleasant.

The present invention seeks to provide a method of rendering colorimages which can be used with palettes which are not well structured,and may be large, without producing transient and pattern jumpingartifacts to which standard error diffusion methods are susceptible.

SUMMARY OF THE INVENTION

In one aspect, this invention provides a method of rendering an image ona display, the method comprising:

-   -   receiving input data representing the color of a pixel to be        rendered;    -   combining the input data with error data generated from a least        one pixel previously rendered to form modified input data;    -   determining in a color space the simplex enclosing the modified        input data, and the display primary colors associated with the        simplex;    -   converting the modified image data to barycentric coordinates        based upon the simplex and setting output data to the primary        having the largest barycentric coordinate;    -   calculating the difference between the modified input data and        the output data for the pixel and thereby generating error data        for the pixel;    -   applying the error data thus generated to at least one        later-rendered pixel; and    -   supplying the output data for a plurality of pixels to the        display and thereby rendering the image on the display.

In one form of this process, the modified input data is tested todetermine whether it is within the color gamut of the display and, ifthe modified input data is outside this color gamut, the modified inputdata is further modified by being projected on to the color gamut. Thisprojection may be effected towards the neutral axis of the color spacealong lines of constant lightness and hue. Alternatively, the projectionmay be effected towards to the color represented by the input data forthe pixel until the gamut boundary is reached. Typically, the colorspace used will be three-dimensional, so that the simplex will be atetrahedron. The error data may, and typically will be, spread over morethan one pixel. For example, if the method of the present invention iseffected using a top-to-bottom and left-to-right order of pixelprocessing, the error data will normally be spread over at least thepixel to the right of, and the pixel below, the pixel being rendered.Alternatively, the error data may be spread over the pixel to the rightof, and the three pixels below and adjacent the pixel being rendered.Especially, in the latter case, it is not necessary that an equalproportion of the error data be spread over all the pixels to which itis dispersed; for example, when the error is spread over the pixel tothe right of, and the three adjacent pixels in the next row, it may beadvantageous to assign more of the error data to the two pixels whichshare an edge with the pixel being rendered, as opposed to the twopixels which only share a vertex.

The present invention extends to an apparatus comprising a displaydevice having a plurality of pixels, each of which is arranged todisplay any one of a plurality of primary colors, and a computing devicecapable of carrying out the method of the present invention andsupplying its output data to the display device, thereby causing thedisplay device to display an image.

The present invention also extends to a non-transitory computer storagemedium comprising instructions that when executed by a processor causethe processor to carry out the method of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 of the accompanying drawings is a schematic flow diagram of aprior art palette based error diffusion method.

FIG. 2 is a schematic flow diagram, similar to that of FIG. 1, butillustrating a preferred method of the present invention.

DETAILED DESCRIPTION

The present invention is based upon the recognition that the transientand pattern jumping artifacts discussed above result from the fact thatthe quantizer (108 in FIG. 1) has available to it an under-constrainedlist of primaries. In a three dimensional color space, any color in thedevice gamut can be rendered by dithering only four primaries, and thepresent invention is based upon constraining the choice of primaries inan appropriate way to ensure that only a restricted set of primaries areused during quantization.

The subset of primaries that can used in a dither pattern to represent agiven color is not unique; for example in a three dimensional colorspace, any set of four or more primaries which define a volume in thecolor space enclosing the given color can be used in a dither pattern.Even if one restricts the subset of primaries to only four, any set offour primaries which define a tetrahedron enclosing the given color canbe used. However, to avoid pattern jumping artifacts, the assignment ofsubsets of primaries to particular colors should be made in such a waythat any parametric path through color space results in a smooth changein proportions of the various primaries used with respect to theparameter. This can be achieved by decomposing the total gamut of thesystem (the convex hull of all the primaries) into tetrahedra withprimaries as vertices and then assigning to each color to be renderedthe subset of primaries corresponding to the vertices of its enclosingtetrahedron. This may be effected by Delaunay triangularization, whichdecomposes the convex hull of the primaries into a set of tetrahedra,the circumspheres of which do not enclose any vertex from anothertetrahedron. This is convenient, but other decompositions of the colorgamut may also be beneficial; for example, to reduce halftonegraininess, the subsets of primaries could be chosen to have lowvariation in lightness. It will be appreciated that the decompositionmethods can be generalized to color spaces of any number of dimensionsby the use of the appropriate simplexes for the numbers of dimensionsinvolved instead of using tetrahedra in a three dimensional space.

A preferred embodiment of the process of the invention is illustrated inFIG. 2 of the accompanying drawings, which is a schematic flow diagramgenerally similar to FIG. 1. As in the prior art method illustrated inFIG. 1, the method illustrated in FIG. 2 begins at an input 102, wherecolor values x_(i,j) are fed to a processor 104, where they are added tothe output of an error filter 106 to produce a modified input u_(i,j).(Again, this description assumes that the input values x_(i,j) are suchthat the modified inputs u_(i,j) are within the color gamut of thedevice.) If this is not the case, some preliminary modification of theinputs or modified inputs may be necessary to ensure that they liewithin the appropriate color gamut.) The modified inputs u_(i,j) are,however, fed to a gamut projector 206.

The gamut projector 206 is provided to deal with the possibility that,even though the input values x_(i,j) are within the color gamut of thesystem, the modified inputs u_(i,j) may not be, i.e., that the errorcorrection introduced by the error filter 106 may take the modifiedinputs u_(i,j) outside the color gamut of the system. In such a case, itwould not be possible to choose a subset of primaries for the modifiedinput u_(i,j) since it would lie outside all defined tetrahedra.Although other ways of this problem can be envisioned, the only onewhich has been found to give stable results is to project the modifiedvalue u_(i,j) on to the color gamut of the system before furtherprocessing. This projection can be done in numerous ways; for example,projection may be effected towards the neutral axis along constantlightness and hue. However, the preferred projection method is toproject towards the input color until the gamut boundary is reached.

The projected input u′_(i,j) values are fed to a simplex finder 208,which returns the appropriate subset of primaries {P_(ks)}, to aprocessor 210, which also received the projected input u′_(i,j) values,and converts them to barycentric coordinates of the tetrahedron (orother simplex) defined by the subset of primaries {P_(ks)}. Although itmight appear that the subset of primaries {P_(ks)} should be based onthose assigned to the input pixel color x_(i,j), this will not work; thesubset of primaries must be based upon the projected input u′_(i,j)values. The output λ of processor 210 is supplied to a quantizer 212,the function of which is very different from that of the quantizer 108shown in FIG. 1. Instead of performing conventional error diffusion, thequantizer 212 chooses the primary associated with the largestbarycentric coordinate. This is equivalent to a barycentric thresholdingwith the threshold (⅓, ⅓, ⅓)(see the aforementioned Arad et al.document), which is not equivalent to the minimum distance determinationcarried out by quantizer 108 in FIG. 1. The output y_(i,j) fromquantizer 212 is then sent to the device controller in the usual manner,or stored.

The output y_(i,j) values, and either the modified input values u_(i,j)or the projected input values u′_(i,j) (as indicated by the broken linesin FIG. 2), are supplied to a processor 214, which calculates errorvalues e_(i,j) by:

e _(i,j) =u′ _(i,j) −y _(i,j) or

e _(i,j) =u _(i,j) −y _(i,j)

(depending upon which set of input values are being used) and passesthis error signal on to the error filter 106 in the same way asdescribed above with reference to FIG. 1.

In theory, it would appear that the error values e_(i,j) should becalculated using the original modified input values u_(i,j) rather thanthe projected input values u′_(i,j), since it is the former whichaccurately represents the difference between the desired and actualcolors of the pixel; in effect, using the latter values “throws away”the error introduced by the projection step. Empirically, it has beenfound that which set of input values is used does not have a majoreffect on the accuracy of the color representation. Furthermore, indeciding whether to use the input values before or after the projectionin the error calculation, it is necessary to take account of the type ofprojection effected by the gamut projector 206. Some types ofprojection, for example projection along lines of constant hue andlightness, provide a continuous and fixed extension of the quantizerdomain boundaries to the out-of-gamut volume, and thus permit the use ofthe unprojected input values in the error calculation without risk ofinstability in the output values. Other types of projection do notprovide both a continuous and fixed extension of the quantizer domainboundaries; for example, projection towards the input color until thegamut boundary is reached fails to provide a fixed extension of thequantizer domain boundaries but instead the quantizer domains changewith input values, and in these cases the projected input values shouldbe used to determine the error value, since using the unprojected valuescould result in an unstable method in which error values could increasewithout limit.

From the foregoing, it will be seen that the present invention canprovide improved color in limited palette displays with fewer artifactsthan are obtained using conventional error diffusion techniques. Thepresent invention may be used in display systems capable of displaying acontinuum of colors (or at least a very large number of colors) but inwhich the available primaries are not evenly spread throughout the colorgamut; for example interference based displays which control a gap widthcan display a large number of colors at each pixel, but with apre-determined structure among the primaries, which lie on aone-dimensional manifold. The present invention may also be used withelectrochromic displays.

For further details of color display systems to which the presentinvention can be applied, the reader is directed to the aforementionedECD patents (which also give detailed discussions of electrophoreticdisplays) and to the following patents and publications: U.S. Pat. Nos.6,017,584; 6,545,797; 6,664,944; 6,788,452; 6,864,875; 6,914,714;6,972,893; 7,038,656; 7,038,670; 7,046,228; 7,052,571; 7,075,502;7,167,155; 7,385,751; 7,492,505; 7,667,684; 7,684,108; 7,791,789;7,800,813; 7,821,702; 7,839,564; 7,910,175; 7,952,790; 7,956,841;7,982,941; 8,040,594; 8,054,526; 8,098,418; 8,159,636; 8,213,076;8,363,299; 8,422,116; 8,441,714; 8,441,716; 8,466,852; 8,503,063;8,576,470; 8,576,475; 8,593,721; 8,605,354; 8,649,084; 8,670,174;8,704,756; 8,717,664; 8,786,935; 8,797,634; 8,810,899; 8,830,559;8,873,129; 8,902,153; 8,902,491; 8,917,439; 8,964,282; 9,013,783;9,116,412; 9,146,439; 9,164,207; 9,170,467; 9,170,468; 9,182,646;9,195,111; 9,199,441; 9,268,191; 9,285,649; 9,293,511; 9,341,916;9,360,733; 9,361,836; 9,383,623; and 9,423,666; and U.S. PatentApplications Publication Nos. 2008/0043318; 2008/0048970; 2009/0225398;2010/0156780; 2011/0043543; 2012/0326957; 2013/0242378; 2013/0278995;2014/0055840; 2014/0078576; 2014/0340430; 2014/0340736; 2014/0362213;2015/0103394; 2015/0118390; 2015/0124345; 2015/0198858; 2015/0234250;2015/0268531; 2015/0301246; 2016/0011484; 2016/0026062; 2016/0048054;2016/0116816; 2016/0116818; and 2016/0140909.

It will be apparent to those skilled in the art that numerous changesand modifications can be made in the specific embodiments of theinvention described above without departing from the scope of theinvention. Accordingly, the whole of the foregoing description is to beinterpreted in an illustrative and not in a limitative sense.

1. A system for producing a color image, the system comprising: adisplay device having a plurality of pixels, each of which is arrangedto display any one of a plurality of primary colors; and a computingdevice in communication with the display device, the computing devicebeing configured to render color images on the display device by:receiving input data representing colors to be displayed at each of theplurality of pixels; combining the input data with error data to createmodified input data; determining in a color space a simplex enclosingthe modified input data, wherein the vertices of the simplex comprise asubset of the display primary colors that is smaller than the completeset of display primary colors, hereafter a simplex primary color;setting output data for a first pixel corresponding to one of thesimplex primary colors; and calculating a difference between themodified input data for the first pixel and the output data for thefirst pixel, thereby generating new error data for the first pixel;combining input data for a second pixel with the new error data for thefirst pixel to create new modified input data for the second pixel;determining in the color space a new simplex enclosing the modifiedinput data and the new modified input data for the second pixel, whereinthe vertices of the new simplex comprise a subset of the display primarycolors that is smaller than the complete set of display primary colors,hereafter a new simplex primary color; and setting output data for thesecond pixel corresponding to one of the new simplex primary colors. 2.The system of claim 1, wherein the computing device evaluates themodified input data to determine whether it is within a measured colorgamut of the display and, if the modified input data is outside themeasured color gamut of the display, further modifies the modified inputdata by projecting the modified input data on to the measured colorgamut of the display to produce projected input data which are used inplace of the modified input data.
 3. The system of claim 2, wherein theprojection of the modified input data is effected towards the neutralaxis of the color space along a line of constant lightness and hue. 4.The system of claim 2, wherein the projected input data is used for thegeneration of the error data.
 5. The system of claim 1, wherein thecolor space is three-dimensional so that the simplex is a tetrahedron.6. The system of claim 1, wherein the new error data for the first pixelis used in the processing of input data for a third pixel.
 7. The systemof claim 1, wherein the display device is an electrophoretic display. 8.The system of claim 1, wherein the display device includes a controller,and the computing device sends the output data for the first pixel andthe output data for the second pixel to the controller.